IME Intern

  • DataDirect Networks
  • Frances Dr, Chatsworth, GA 30705, USA
  • Mar 08, 2018

Job Description

DataDirect Networks

Job ID 2018-2153
Job Locations FR-MEUDON LA FORETJob Posted Date 18 hours ago

IO component of OpenMPI over Burst Buffer

DDN Storage is the world leader in high performance and massively scalable data management and storage solutions that accelerate business results and scientific insights for data-centric organizations worldwide. Our unified, end-to-end platform niquely addresses the tiered storage and large scale data management demands of mixed workloads, multiple collaborative data centres and Web and cloud environments. Across traditional and commercial high performance markets, customers rely on DDN Storage to solve the most demanding big data problems in industries such as cloud, online content and social networking, security and intelligence, life sciences, finance, energy and media production.

DDN's Infinite Memory Engine (IME) is a scale-out, software-defined, flash storage platform that streamlines the data path for application I/O. IME interfaces directly to applications and secures I/O via a data path that eliminates file system bottlenecks.

In order to widespread the adoption of its new generation of Burst Buffer technology to the end-users, DDN takes part of open-source projects to include optimized accesses to its IME product. The objective of the mission is to modify OpenMPI - i.e., a high performance and straightforward message passing library ( - to allow a direct access to the IME servers using a set of native functions exposed by IME.
The internship will start with a learning period on the burst buffer technology and the IME product. With the help of DDN team, the intern will then modify the IO component of OpenMPI (called OMPIO) to include existing functions from the IME native interface and allow fast IO path to IME servers.

The internship will ensure that the new code:
1/ reaches the expected level of performance;
2/ is sufficiently stable to support the execution of large-scale applications;
3/ gets reviewed and merged into the OpenMPI's upstream repository on Github.

Finally, (and optionally, depending on the time we have), the internship will include the two additional points:
1/ Porting some existing code from the legacy IME native interface to the new one.
2/ Extending the existing IME native interface with new systems calls (e.g., readdir, mkdir, unlink, ...).
DDN R&D team will be providing support all along the way.

The mission will consist of:
  • Understand the IME product functionalities;
  • Modify the source code of OpenMPI (IO component -- OMPIO) to allow fast IO path to IME servers through the IME native interface;
  • Set the infrastructure for performance and stability testing;
  • Provide a performance and stability report of the new code;
  • Ensure the new code to OpenMPI gets reviewed by the OpenMPI maintainers and merged to upstream OpenMPI repository on Github;
  • Optional 1: porting some existing code from the legacy IME native interface to the new one.
  • Optional 2: extending the existing IME native interface with new systems calls.

Diplomas / Training Minimum Master 1 in Computer Science or 2nd year in engineering school

Required skills
- C development and system programming
- Version control development (i.e. Git)
- UNIX / Linux environment
- Scripting language: Shell or Python

Skills / Qualities appreciated
- Distributed programming with MPI (Message Passing Interface)
- Benchmarking in distributed environments
- Basic knowledge of High Performance Computing

Start date As soon as possible

Duration 3-4 months