Junior Data Engineer
- Design, develop and modify software systems, using scientific analysis and mathematical models to predict and measure outcome and consequences of design
- Develop individual modules, test the functionality and integrate it with the main software build, Perform unit testing and functional testing of the product
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Understand existing software solutions if any and enhance the design as per requirements
- Understand the existing data model of various databases or any other data sources already deployed from the perspective of software development for a new and improved system
- Build technical solutions required for optimal ingestion, transformation, and loading of data from a wide variety of data sources using SQL and AWS technologies
- Integrate the data sources into the application, perform desired operations as per requirements and output the data to other sources as desired
- Perform thorough testing, profiling and validation to support the accuracy of data transformations and data verification used in the machine learning models
- Analyzing information to recommend and plan the installation of new systems or modifications of an existing system
- Analyze complex data systems and elements, data flow, relationships and dependencies to contribute to conceptual, logical and physical data models
- Understand and formulate processing pipelines of large, complex data sets that meet functional / non-functional business requirements
- Create and maintain optimal data pipeline architecture
- Maintenance and Bug fixes post deployment of the Software Product
- Store, retrieve, and manipulate data for analysis of system capabilities and requirements