Platform

New Features

1. Migration: The user can seamlessly migrate the Data Preparation component along with its associated Pipelines. This feature greatly facilitates transferring pipelines from one environment to another, ensuring a smooth transition between different environments.
2. Data Center:
1. Data Stores:
1. We have introduced a convenient Refresh option for the Sandbox file-based Data Stores, enabling users to easily update and reflect the latest changes made to these data files.
2. The Data Store List Page has been enhanced with the addition of a user-friendly Information Drawer UI for improved data store management. This intuitive interface provides quick access to important data store details.
2. Optional LOV Filters for Data Store Metadata: LOV (List of Values) filters in the Data Store Metadata have now been made optional. Users can choose to enable or disable these filters using checkboxes in the user interface, providing greater flexibility and control over filtering options based on specific requirements.
3. PostgreSQL Integration in Data Store Settings: As part of the Admin Settings, we have introduced the option to configure and utilize PostgreSQL as a Data Store.

Please Note: The BDB Platform is compliant with OWASP security standards.

Data Preparation

New Features

1. New Transforms (under the Columns category):
1. Add Blank Column
2. Change to String Data Type: It can change data from Int/Float/Date to string.
3. Relocate of Column Transformation
4. Merge Column Transformation: Merging of two or more columns to create a new column
5. Sorting Transformation
6. Delete All Transformation
7. Delete Except/ Keep Columns
2. Column Visibility Control: We have introduced a convenient Show/Hide option for columns displayed in the Data Grid. This feature empowers users to customize their data viewing experience by selectively displaying or hiding specific columns based on their preferences and analysis requirements.
3. The column header displays the following information based on the column types:
1. Columns with Integer values- the Min and Max values
2. Columns with String values- Total unique count or no. of categories
3. Columns with Date values- Range of dates including the min-max date
4. Enhanced Data Preparation Page Information: At the bottom of the Data Preparation page, we now display key metrics to provide valuable insights into the dataset being analyzed. These metrics offer essential contextual information, enabling users to make informed decisions, perform data profiling, and gain a deeper understanding of the dataset being prepared.
   This includes:
1. Column Count: The total number of columns in the dataset, allowing users to quickly assess the complexity and scope of the data.
2. Row Count: The total number of rows in the dataset, providing an overview of the dataset's size and volume.
3. Data Type Count: The number of distinct data types present in the dataset, enabling users to understand the variety and diversity of data formats and structures.

Data Pipeline

New Features

1. Python Version Upgrade: We have upgraded the Python version to 3.10, incorporating the latest enhancements and features.
2. Kafka Library Update: We have updated the Kafka library for our Python components, ensuring seamless integration and enhanced functionality with Kafka messaging systems. This update enables smoother data streaming and messaging capabilities, facilitating real-time data processing and analysis.
3. Introduction of DB Sync as a Failure Event: We have introduced DB Sync as a failure event, allowing for automated synchronization in the event of a pipeline failure. This ensures data consistency and integrity, minimizing any potential disruptions caused by pipeline failures and maintaining data accuracy throughout the data processing workflow.
4. Failure Alert for Pipeline Notifications: To improve transparency and response time, we have implemented a Failure Alert mechanism that notifies users promptly in the event of a pipeline failure. This alert ensures that relevant stakeholders are immediately informed, enabling timely troubleshooting and resolution of issues.
5. Jobs:
1. Introducing Python Job: We have introduced Python Jobs, providing an expanded range of job execution capabilities.
2. Job Run History and Summary: We have introduced features that capture job run history and provide summary information to track job execution details, analyze performance metrics, and gain insights into job outcomes.
6. Single Page Configuration (SPC) for the Data Pipeline Module: We have implemented a Single Page Configuration (SPC) for the Data Pipeline module by consolidating all necessary settings and options on a single page. This simplified user interface improves usability, reduces navigation complexity, and enhances the overall efficiency of pipeline configuration tasks.

Enhancements

1. Jobs
1. Dynamic Status Change in Jobs: We have introduced dynamic status changes in Jobs, allowing for real-time updates and improved visibility into the progress and status of job executions.
2. Tooltip for Meta Information tab in Tasks: To enhance user experience and provide additional context, we have added tooltips to the Meta Information tab for tasks. These tooltips offer helpful information and explanations, enabling users to better understand and interpret task details within the Jobs module.
2. Collapsible UI for the Advanced Logs: We have implemented a collapsible user interface (UI) for the Advanced Logs tab. Users can now expand or collapse logs as needed, facilitating efficient troubleshooting and analysis of log information.
3. Enhanced Pipeline Monitoring option: We have made significant improvements to the Pipeline Monitoring option. These enhancements empower users to effectively monitor and manage their data pipelines.

Data Science Lab

New Features

1. Data Science Lab Models: Implemented custom scripts for the seamless saving, loading, and prediction of Data Science Lab models. This empowers our team to come up with more efficient data models with improved accuracy and productivity.
2. Auto ML Experiments
1. Enhanced AutoML Model Training Process: We have introduced a dynamic Jobs system to support the AutoML model training process, replacing the previous static container implementation.
2. Improved Model Explainability: As part of the same Job, Shap values will now be automatically generated, providing valuable insights into the factors influencing our models' predictions. These insights can be visualized using the Explainability dashboard.
3. Streamlined Job Management: Once the training process is completed, the associated Jobs will be automatically terminated, optimizing resource allocation, and ensuring efficient utilization of computing resources.
3. Introduction of Working Directory:
1. We have introduced a user-friendly option in the interface that allows users to create new folders and files directly within the Notebook page for better collaboration and easy access.
2. Files can now be uploaded directly into the Data folder located within the respective Projects. This centralized location ensures that data files are conveniently stored alongside the relevant project, simplifying data management.
4. Algorithms: We have introduced unsupervised Forecasting and Natural Language Processing algorithms within the Data Science Lab Notebooks. These cutting-edge algorithms empower us to uncover valuable insights from the most complex datasets.

Please Note: The above-listed Algorithm types are modified in the Administration module (in the Data Science Lab Settings) as well as Project level settings (inside the Data Science Lab module).

5. PySpark Environment:
1. Expanded support for Data Sets as readers within the PySpark Environment.
2. Introduced comprehensive support for Data Writers within the PySpark Environment. This enhancement can enable our team to efficiently write and save data ensuring seamless integration with downstream processes.

Please Note: The supported Data Sets and Data Connector types for the Data Set readers and Data Writers in the PySpark Environment are MySQL, MSSQL, Oracle, MongoDB, PostgreSQL, and ClickHouse.

Enhancements

1. Progress Bar: Custom implementation of the Progress Bar inside the Data Science Notebook.

Designer

New Features

1. Decomposition
1. Streamlined Dimension Selection: Hierarchy has been eliminated from the selected dimension, simplifying the data decomposition process, and improving usability.
2. Enhanced Series Field Indicator: Users can now assign conditional colors to series values, enabling more intuitive visualizations and highlighting important insights.
3. Animated Visualization: Animation functionality has been added as a configurable property, allowing users to incorporate dynamic visual effects into their data presentations.
2. Grids: Users now have access to a Range Indicator in the form of a bar alert type for both normal and dynamic ranges in Grids. This feature provides visual cues to quickly identify data points falling within or outside specified ranges, aiding in efficient data analysis.
3. Filter Chips: Users have the flexibility to modify the titles associated with SVG images within Filter Chips, enabling them to provide more descriptive and meaningful labels.
4. Datasheet: The Datasheet module has undergone updates to enhance script execution capabilities and introduce user interface changes for seamless pagination.

Enhancements

1. Timeline chart:
1. Improved Slider Behavior: When maximizing or minimizing the chart, the position of the slider remains fixed, ensuring continuity and preserving the user's selected timeframe.
2. Enhanced Right Axis Font Control: Users can customize font properties for the right axis in the Timeline chart, allowing for better readability and design consistency.
2. Map component:
1. Custom Tooltip Formatter: Introduced a feature that enables users to apply custom formatting to tooltips in the Map component, facilitating clearer and more tailored information display.
2. Indicator Support for Series Field: Users can now leverage indicators to enhance the visual representation of the Series field in the Map component, providing additional insights and highlighting key data points.
3. Color Range Alert Pop-up: The Color Range Alert pop-up has undergone user interface improvements and validation enhancements to ensure a seamless and error-free experience when defining color ranges.
4. Tab Component:
1. Tab Width Control: A new property has been added to allow users to control the width of all tabs in the Tab Component, providing greater flexibility in designing and organizing dashboard layouts.
2. Height Adjustment Based on Tab Selection: The height of the dashboard now automatically adjusts based on the Text & selected icons. Additionally, the Tab Component now supports script execution based on tab selection, allowing for dynamic content updates.
3. Vertical Alignment and Tab Border Property: Users can now customize the vertical alignment of tabs and choose to display borders when selecting a tab.
4. Tab Content Alignment and Associate Groups UI Enhancements: The alignment of Tab content has been improved for a more polished appearance, and user interface enhancements have been implemented to enhance the usability and clarity of the Associate Groups feature.

Story

New Features

1. Data Store Settings: PostgreSQL compatibility has been added as a Data Store option, allowing users to create engaging Stories using PostgreSQL.
2. File Upload Guidelines: Clear guidelines and best practices have been established for uploading files to create Stories, ensuring a standardized approach and optimal results.
3. Global Filter: A convenient and prominent Filter Panel has been integrated into the Story module, enabling users to easily apply global filters to refine and customize their storytelling experience.
4. Calculated Field: The user interface for PostgreSQL now includes seamless implementation of calculated fields, empowering users to perform advanced calculations and derive valuable insights from their data.
5. Data Store Jobs:
1. Batch Query Enhancement: Improved performance and efficiency have been achieved by implementing batch queries for Data Store Jobs, allowing for faster execution of multiple queries.
2. Job Synonyms: Job-related terminology has been updated with synonyms to enhance clarity and understanding throughout the application.
6. Grid chart: Users now have the flexibility to control the column width in Grid charts, enabling them to customize the visual representation of their data according to their specific requirements.

Enhancements

1. Gauge chart:
1. Target Value: Select any column in Gauges to map as Target Value.
2. Added Number formatting in the Gauge chart.
3. Enhanced the visual appeal and functionality of the Gauge chart by leveraging the capabilities of the Designer module to update its design elements and properties.
2. Optimization of the backend service calls to reduce instances of redundant service calls & enhance performance across Edit, Analyze, and View modes.
3. Pie chart: Introduced a property to display both the actual value and percentage in the Pie chart enabling comprehensive insights and facilitating better data interpretation.