DHTS IT ANALYST - Deidentification Data Engineer
Job Title: IT Developer II (Deidentification Data Engineer)
Primary Role: This position reports to the DHTS-Data Partnerships Director of Data and Analytics Platforms. This position will be responsible for the management and execution of the deidentification processes applied to the Duke University Health System data assets to be included in the Federated Clinical Applications Platform (FCAP), and the management and administration of the FCAP deidentification cloud environment.
The position will be a member of the Data Partnerships Data Engineering team and will additionally provide expertise in the development of data integration and delivery pipelines to deliver new data modalities into the FCAP and Duke’s Data Lake. These solutions will capitalize on technologies to improve the value of analytical data, improve the effectiveness of information stewardship, and streamline the flow of data in the organization. Solutions will focus on using state-of-the-art data and analytics tools including traditional and near real-time data warehousing, big data, relational and document-based databases using both extract, load, and transform (ELT) toolsets as well as REST APIs and FHIR. The ideal candidate will be comfortable with data science platforms with proven experience leveraging DevOps and Automation/Orchestration tools.
• Create and follow defined procedures in the deidentification of patient medical information
• Maintain and tune the deidentification environment to perform optimally and comply with DUHS and DHTS policies and standards
• Collaborate with the Duke partner on improving the deidentification programs and processes, and work with the partner and the Duke Cloud Team on troubleshooting issues
• Assemble large, complex data sets that meet business requirements
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Recommend design of analytics solutions that improves data integration, data quality, and data delivery with an eye toward re-useable components
• Create and maintain an optimal data pipeline architecture
• Articulate differences, advantages, and disadvantages between architectural solution methods
• Work with Agile team members to document and execute test plans for data loading and data validation scripts. Support the code promotion process through development and production as required by using standard CI/CD processes
• Develop, implement, and maintain schedule/dependency logic for automated ETL processing
• Develop monitoring, logging, and error notification processes to ensure data is updated as expected and processing metrics reported
• Participate in the creation and maintenance of standards for coding, documentation, error handling, error notification, logging, etc.
• Accountable for conforming to established architectural, developmental, and operational standards and practices including the creation of metadata
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics
• Work with stakeholders including the Executive, Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs
Education: Bachelor's degree in a related field, or four years of equivalent technical experience required
Required Experience: We are looking for a candidate with 5+ years of experience in a Data Engineer role who should also have hands-on, professional experience with the following:
• Relational SQL and NoSQL databases
• Writing and executing Python programs and shell scripts on Linux
• Intermediate Linux administration
• Data Engineering on Microsoft Azure
• Data pipeline and orchestration tools such as Azure Data Factory and SQL Server Integration Services
• Developing on cloud-based analytic platforms such as Azure Synapse
• Performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
• Supporting and working with cross-functional teams in a dynamic environment
• A successful history of manipulating, processing, and extracting value from large disconnected datasets
• Intermediate to advanced skills in Python programming
• Intermediate to advanced skills in the Azure Cloud Data Engineering Stack
• Intermediate Linux administration and shell scripting
• Advanced working SQL knowledge and experience working with relational and non-relation database systems
• Strong analytic, documentation, and organizational skills
• Prior experience in health care IT
• Working knowledge of Azure DevOps & Automation/Orchestration
• Knowledge of open-source software solutions and open-source as a business model
• Technical breadth across application development, enterprise architecture, or application integration
• Understanding of Agile methodology
• Knowledge of APIs, API Integration, and API Management
The information above describes the general nature and level of work assigned to this position. It is not intended to be an exhaustive list of all duties and responsibilities required of position incumbents.
Duke is an Affirmative Action/Equal Opportunity Employer committed to providing employment opportunity without regard to an individual's age, color, disability, gender, gender expression, gender identity, genetic information, national origin, race, religion, sex, sexual orientation, or veteran status.
Duke aspires to create a community built on collaboration, innovation, creativity, and belonging. Our collective success depends on the robust exchange of ideas—an exchange that is best when the rich diversity of our perspectives, backgrounds, and experiences flourishes. To achieve this exchange, it is essential that all members of the community feel secure and welcome, that the contributions of all individuals are respected, and that all voices are heard. All members of our community have a responsibility to uphold these values.
Essential Physical Job Functions: Certain jobs at Duke University and Duke University Health System may include essentialjob functions that require specific physical and/or mental abilities. Additional information and provision for requests for reasonable accommodation will be provided by each hiring department.
Explore Duke Health Technology Solutions
Want to see where you’d be working? Check out this location and everything it has to offer you.
Serving With Purpose
You’ve always been called to care – and we share your dedication to making a meaningful difference in the community. With the entire team united through this purpose, there are no limits to what we can achieve together. You’ll have all the resources you need here to leave a legacy of exceptional service because at Duke Health we support you in your calling to care.
Awards and Recognition
Duke Health is consistently recognized as a top employer for our dedication to the team members and the patients we serve.
On Forbes list of Best Large Employers, we are the #1 healthcare recipient in North Carolina for 2023.
Our hospitals are consistently recognized as LGBTQ Healthcare Equality Leaders by Healthcare Equality Index.
We are among about 9 percent of U.S. hospitals that hold the American Nurses Credentialing Center’s Magnet Recognition honor.
CHIME Digital Health Most Wired recipient since 2018, honoring our ability to elevate the health and care of communities around the world.
We are proud to be named as the #1 healthcare system in North Carolina on Fortune’s America’s Most Innovative Companies 2023 list.
Jobs For You
As you search for opportunities that fit your skills, any jobs you view will appear here.
You haven’t saved any jobs yet, but when you find an opportunity that seems right for you, bookmark it so you can return to it easily.
More About Duke Health
Upcoming Hiring Events
Check out our calendar of events to stay up-to-date on information sessions, webinars, and other events featuring new opportunities at Duke Health.
Benefits You’ll Love
We want to make your life better both in and out of work. That’s why we offer comprehensive benefits and unique perks that cater to you and your family.
What We Stand For
Our values define the work we do. Team members here share in these values, creating a community that is connected in purpose and committed to excellence.
With hospitals in Raleigh and Durham and over 140 specialty and primary care clinics around the Triangle region, there are many options to join Duke Health in the city and the suburbs.