Platform

1. Data Center:
1. New Data Connectors:
1. Arango Connector
2. Elastic Connector
2. Data as API: the Delete option for the Data as API list.
3. Data Preparation: The Data Preparation option has been added for the Data Sets and Data Sandbox. (The supported RDBMS data sets for the Data Preparation option are MySQL, MSSQL, MongoDB, Oracle, PostgreSQL, and Snowflake)
4. Data Connector Enhancements:
1. Multiple IP support with High Availability has been provided for the Postgre SQL and MongoDB connectors.
2. Snowflake Connector: Added a new optional field Warehouse in the connector configuration page.
3. Hive Connector: Added a new optional field Database Name in the connection configuration page.
4. Mongo DB: Provided the Authsource configuration for mentioning a different Authentication DB.
2. Admin Module:
1. Introduced the Core Ownership Transfer to securely handle data transfer from one user to another user.
2. Provided the Validate option for connection validation for the Datastore Settings option (for MongoQL and Elastic options).
3. Enhancements for the Document Management.
1. It displays a list of documents by User & Use Group.
2. It is now possible to unpublish a Dashboard from the User Group list of documents.

Please Note:

1. The BDB Platform is compliant with the OWASP security standards.
2. The Elastic connector and Data Preparation option are Beta releases.

Data Pipeline

1. Spark/Scala/ Kafka upgrade [All the components will run on the below versions of Scala and spark with the support of the below Kafka version]:
1. Upgraded Spark version to 3.3.0
2. Updated Scala version to 2.12.12
3. Upgraded Kafka version to 3.1.0
2. Writers: HA (High Availability) support for:
1. MSSQL
2. Oracle
3. ClickHouse
4. PostgreSQL
5. MySQL
6. MongoDB
3. Readers: HA (High Availability) support for:
1. MSSQL
2. Oracle
3. ClickHouse
4. PostgreSQL
5. MySQL
6. MongoDB
4. Provided HA(High Availability) support for Enrichment component.
5. Improved writer performance by including schema definition.
6. ClickHouse Support:
1. Reader support
2. Writer support
3. Enrichment support
4. DB Sync support.
7. SQL [Batch & Aggregation]: The SQL Component now has an option aggregation function on complete streaming data that has been processed by the component.
8. Kafka Producer and Consumer Enhancement:
1. Introduced a Kafka Producer component to connect to external Kafka topics for different applications.
2. Kafka Consumer supports SSL Security Type with Host Aliasing.
9. Intelligent Scaling: A feature to scale your component to the max instance to reduce the data processing lag. This feature detects the need to scale up the components on higher data traffic and reduces the instances if there is no data traffic.
10. Connection Validation: Provided this option to validate all the connections before activating a pipeline. E.g., if there is an RDBMS reader, we can click on the Validation icon to validate it.
11. Data Channel enhancement: Introduced more broker-level insights like no. of partitions and disk usage.
12. Introduced a Cluster event page to monitor the Kafka brokers and topics related to pipelines. Even the deleted pipeline topics will list there.
13. Spark Mongo Component: Introduced Connection String, HA Mongo option, and Spark Schema upload to reduce Write Operation.
14. Implemented required changes to align with the new version of the DS Lab module that supports Data Preparation and Auto ML options.

Data Science Lab

1. Auto ML Integration:
1. There will be an option to train models with different algorithms and get the best model among them using Auto ML applications.
2. A new tab along with datasets is provided at the project level for Auto ML experimentations.
2. Data Preparation Integration:
1. Integrated the Data Preparation services in the DS Lab module.
2. Provided an icon in the Datasets tab to generate Data Preparations.
3. Pytorch inferencing in Pipeline:
1. A Pytorch model can be trained, saved, and registered inside the Data Pipeline module.
2. The registered model can infer with the test data and get the predictions in the DS Lab model runner.
4. Data Catalog Implementation:
1. Displays the consolidated details of Data science projects, Notebooks, models, and Utility scripts.
2. Provided illustrative representation in UI.
5. Kernel Session Manager: Introduced a Kernel Restart icon on the Notebook header panel to kill the current session and create a new one.
6. The Output cell height has been increased to 500px.
7. Ellipsis Icon for Cell Operations: Provided the cell operations inside the Ellipsis icon to enhance the UI experience.
8. Timestamp for Notebook and Model lists: Provide Timestamp as a separate entity in the list to identify the last updated time from UI.
9. Algorithms:
1. To remove the extra Print statements and unwanted output parameters.
2. Introduced optimized and standardized code.
10. Merge and Split cells: Two or more cells get split or merged inside the DS Lab Notebook page.
11. Static Update of the List: Instead of refreshing the entire list, a loader appears at the entity level to refresh it.
12. Save As Operation: The Save As option has been provided in the Notebook toolbar to create a copy of the DS Notebook.
13. Artifacts enhancement: Provided a Preview option in the Artifacts UI with limited rows. In case of the images, the .png file itself can be previewed.
14. Data Sets Enhancement: Provided a Preview option in the UI with limited rows to preview the added data set.
15. Provided an Add option inside the Notebook to directly add Data Sets and Data Sandbox files.
16. Variable Explorer: Provided a variable explorer to show the name, type, length, and sample values for all variables that are created in the Notebook.

Please Note: The Auto ML is an alpha release.

Designer

1. Datasheet component Enhancement:
1. Provided the Add Row (+) option, the new row gets added to the last.
2. Ability to freeze columns.
3. Provided pagination with the property.
4. Able to edit the frozen column in a newly added row.
5. Introduced the ‘Delete row’ option with a confirmation message before a row gets deleted.
6. Able to retrieve more records of data.
7. The Save option gets updated in red color when something is edited in the datasheet.
8. Updated the JExcel DataGrid and JSuites library with the latest stable version(4.6.1).
2. Filter Chips supports: The supported filter components are:
1. List Filter
2. Radio Filter
3. Heirarchical Filter
4. Combo Filter
3. Leaflet:
1. Implemented indicators with marker icons and color.
2. Support for the maki-markers and glyph-icons in Trip and Trial geometry.
3. Line type & thickness configuration in Trip and Trial geometry.
4. Circle geometry with markers implementation.
5. Provided default Zoom option based on available data points.
4. Grids:
1. Simple Grid : Column-wise font-properties.
2. Size control for the Alert icons.
3. Paging-grid: Provided variables for controlling headers.
5. Mobile/Tablet View: the updated Date picker calendar pop-up for the mobile and tablet views.
6. Export (Chart export): Legend also gets exported in PDF when associated with a chart.
7. Dashboards manage page: Provide the details about the dashboard in the info button.
8. Trellis Chart: Implemented data labels.
9. Script update to implement the following as Properties or SDK Methods:
1. Timeline slider height ratio as property.
2. Timeline slider width control based on the number of records to be shown.

Story

1. New Charting Components:
1. Spider
2. Waterfall
2. Dissolution Tree: Chart Properties are updated with the standard BizViz font type.
3. A Data Profiling option has been introduced:
1. For Dimension: Total Unique Count, Total Duplicate Count, Top 5 based on decreasing order of the Duplicate Count.
2. For Measure: Total Duplicate Count, Minimum, Maximum, Average, Sum, Variance, Standard Deviation, and Top 5 based on decreasing order of the Duplicate Count.
3. For Time- Minimum Date and Maximum date.
4. UI Updates for Charts:
1. The chart padding is updated from the Left, Right, and Bottom sides.
2. Provided the customizable Legend Font Size option.
5. Export Option: The Export as JEPG Image and Export as PNG Image options are provided in the Analyze mode for a view.
6. Distinguish between Datastores:
1. Distinguish between instances and scheduler-based data stores.
2. Distinguish between the Active and Expired instances.
7. Formula Field Editor enhancements:
1. Provided a Search option to look for the drop-down for fields, Operations, and Functions.
2. Provided a sample on how to use Formula Editor in the available space in the Formula Editor.
8. Enhanced the UI experience of the Create New Story option.

Deployment

Enhanced CI/CD process in Deployment
1. Implemented flux cd for keeping Kubernetes clusters in sync.
2. Introduced Harbor for container registry and Helm repository.
3. Updated Helm chart for deployment automation.