Data Services Platform for the Department of Education

Overview of the Engagement

The NYC Department of Education (DOE) encountered a significant challenge with managing attendance and learning data. With the onset of remote instruction during the COVID-19 pandemic, reliance on remote learning tools intensified the issue. Prior to engaging with Spruce, DOE faced data fragmentation across on-premises and cloud systems, making unified reporting a daunting task. Each school used its own methods for tracking student data, impeding the DOE leadership’s ability to gain insights from the collective data. Additionally, data federation, normalization, security, scalability, and cost management were intimidating hurdles in achieving a unified data solution. Spruce’s task was to address these challenges and create a centralized data platform that would revolutionize the way DOE accessed and utilized its data.

Approach

Data Consolidation: Spruce initiated the project by consolidating data from various sources, including learning management tools such as GSuite, D2L, and Teams. This included attendance data, student data, and location data, all integrated into one central data store.

Data Hub for Consolidated Data Access: We created a Data Hub to expand data accessibility to over 1200 schools and external public users. Initially focused on learning data, the project expanded to include device and support data. This centralized view allows schools to access data through reporting tools.

Customized Solution: Spruce customized the platform to meet DOE’s specific needs, empowering them to view school data and activities in new and creative ways.

Long-term Vision: Working closely with DOE leadership, Spruce adapted the architecture, scalability, governance, and security best practices to accommodate not only the current scope but also future expansions, such as HR data, transportation data, and special education data.

Azure Synapse Analytics: Azure Synapse Analytics Implementation: Spruce developed a robust Data as a Platform (DaaP) solution using Azure Synapse Analytics. It processes over 400 data points and manages more than 9 billion records, providing DOE with a secure, scalable, and flexible data services platform.

Continuous Improvement: Spruce and DOE maintained a close partnership with Microsoft to stay aligned with product updates, enhancing the capabilities of Synapse with Databricks, Decision Intelligence, Rules Management, AI-driven data classification, Data Explorer, and cost management tools to optimize Azure Synapse Analytics spending.

Key Metrics

Data accessibility was extended to 1200+ schools
and external users.

Conclusion

By collaborating with Spruce, the NYC Department of Education has achieved an exceptional transformation in its data management and accessibility. The creation of a Data as a Platform (DaaP) using Azure Synapse Analytics and the Data HUB has not only centralized data but has also revolutionized the way the DOE leverages information to make informed decisions. Today, over 1200 schools and external users can access critical information securely and efficiently.

What sets this solution apart is its forward-looking nature. The DaaP solution built by Spruce is not merely a point solution; it has become central to the DOE’s data operations. The DaaP’s impact goes beyond streamlined data management—the scalability and adaptability of the platform ensures that it can support future data needs, including HR, transportation, special education, and beyond. The DaaP solution has evolved into a central asset for the agency, and its potential for further expansion is limitless, serving as a model for data-driven excellence in the public sector. Spruce is proud to be part of this transformative journey, shaping the future of education through data. This solution can be expanded and adapted by other agencies across the city to drive similar outcomes.