We are heading towards a world of greater segmentation, with lawmakers seeking to adopt or strengthen cross-border data flow restrictions. The EU is an interesting example. New data regulations such as the Data Governance Act or the Data Act make it clear that data restrictions could apply to non-personal data, e.g., non-personal data held by public sector bodies. 

The European Parliament proposed to introduce a data localisation provision in the European Health Data Space regulation. The US is also changing position as illustrated by the recent step-back of the U.S. Trade Representative on data localization in the context of free trade negotiations.

In the coming year, resources will be allocated in the following areas:

• Data Governance and Compliance: Establishing data policies, setting in place data classification and access controls, and ensuring compliance with relevant international, regional, national, and state laws like GDPR, CCPA, and so on.
• Data Quality and Data Integration: Data teams may prioritize efforts to improve data quality and ensure accurate and reliable data.
• Advanced Analytics and Machine Learning: Data teams may allocate resources to enhance their capabilities in advanced analytics and ML.
• Data Security and Cybersecurity: Given the increasing frequency and sophistication of cybersecurity threats, data teams may prioritize resources to strengthen data security measures.

In the coming year, IT budgets will be reprioritized for investments in platform consolidation.

These budgets will remain relatively flat in 2024, but will be reprioritized towards investments in consolidating capabilities into fewer platforms. 

Despite the mandate to do more with less, given tough macroeconomic conditions, organizations can’t and won’t halt investment in IT. But, where it may have taken 10s of vendors to accomplish your data platform ambitions in the past, it will take far fewer in the next year and beyond. Investments will focus on just the critical building blocks that form the foundation of a modern data stack.

As we look towards 2024 and beyond, a significant shift is becoming apparent in the architectural paradigms that underpin data strategies across various industries. The historical three-tier architecture, with its clear demarcation between data and application layers, is giving way to a more integrated approach. This trend, which I refer to as “data-centric architecture,” is gaining momentum, particularly with the advent of platforms like Snowflake and the innovative Snowflake Snowpark Container Services.

This integration of applications with data services represents a fundamental change in how we approach data infrastructure. By bringing the applications closer to the data, we not only reduce latency and improve performance but also open up new possibilities for real-time analytics and decision-making. This is especially pertinent in industries where speed and data freshness are critical, such as financial services, healthcare, and retail.

However, this convergence also necessitates a more sophisticated approach to governance and security. As developers and operations teams work more closely with the data layer, the principles of DevSecOps become increasingly important. We must ensure that security and compliance are not afterthoughts but are integrated into the entire lifecycle of data-driven applications. This will likely lead to the development of new frameworks and policies that are tailored to this integrated architecture, ensuring that governance is maintained without stifling innovation.

Currently, the active leadership of the security within AstrumU is driven by our Principal Security Officer who reports into the CTO. The simple truth is that governance and security will continue to grow in support of the business strategy, simply because everyone wants to know that their data is being handled appropriately.  

As we see more emphasis on the use of data, particularly within AI services, there is going to be a growing demand for transparency and accountability on how data is managed and processed through our platform. Our hope is that we can find the balance between innovation versus being held captive by regulatory constraints. The latter option will only benefit the largest organizations with the potential for less visibility and ownership of one’s data, and this will not be to the benefit of society in the long term.

In the next 12 months, we are likely to prioritize resources and focus on several key areas. Here are some of them:

• Data Governance and Compliance: With increasing regulations around data privacy (such as GDPR or CCPA) and a growing emphasis on ethical use of data, data teams will allocate resources to strengthen data governance frameworks. This includes ensuring proper data classification, access controls, consent management, and auditing mechanisms to maintain compliance.
• Data Quality Assurance: As organizations rely more heavily on insights derived from data analysis, the accuracy and reliability of that data become paramount. Data teams will invest in tools and processes for comprehensive quality assurance checks including validation techniques, anomaly detection algorithms, automated cleaning procedures, etc., to ensure high-quality datasets.
• Advanced Analytics & AI/ML Capabilities: To extract actionable insights from vast amounts of structured/unstructured datasets effectively; data teams may focus on developing advanced analytics capabilities such as predictive modeling/machine learning algorithms,text mining/natural language processing (NLP), computer vision/image recognition etc.,to enable intelligent decision-making across various business functions.
• Data Security & Privacy Enhancements: The rising frequency of cyber threats necessitates increased attention on securing sensitive information within the organization’s ecosystem. Data teams might prioritize allocating resources towards enhancing cybersecurity measures by implementing robust encryption protocols, data masking/anonymization techniques, vulnerability assessments, predictive threat intelligence systems etc., that can help safeguard against potential breaches or unauthorized access attempts.