SLURM and Parallel Processing in Jupyter-Lab via OOD¶

When working in Jupyter Lab, you can leverage session outputs or run parallel tasks, which is beneficial for efficient computation across multiple nodes. This guide shows way to run SLURM tasks from a Jupyter notebook using Open OnDemand (OOD).

Accessing OOD¶

Open this link to start a new OOD Jupyter Lab session.
Start a job with 2 nodes and 256 cores.

A window with Jupyter Lab will open, where you can continue working.

Setting Up Jupyter Lab¶

Once Jupyter Lab opens, select File → New Launcher.
Select Notebook → Python 3 (ipykernel) to create a new notebook.

Inserting the Sample Code¶

Paste the following sample code into your new notebook to run a parallel task using MPI:

import ipyparallel as ipp

def mpi_example():
    from mpi4py import MPI
    comm = MPI.COMM_WORLD
    return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}. host={MPI.Get_processor_name()}"

# Request an MPI cluster with 256 engines
with ipp.Cluster(controller_ip="*", engines="mpi", n=256) as rc:
    # Get a broadcast_view on the cluster which is best suited for MPI style computation
    view = rc.broadcast_view()

    # Run the mpi_example function on all engines in parallel
    r = view.apply_sync(mpi_example)

    # Retrieve and print the result from the engines
    print("\n".join(r))
# At this point, the cluster processes have been shutdown

Explanation of the Code¶

ipyparallel: Used for parallel computations with multiple processes.
mpi4py: A library for working with MPI (Message Passing Interface), which is a method for efficient parallel processing across multiple nodes.
The function mpi_example() returns information about each process, such as its rank, total number of processes, and host name.
Using ipp.Cluster, we request a cluster with 256 cores and run the function on all nodes concurrently.

Job Output¶

After Execution¶

When the code is executed, you will see an output similar to this

Starting 256 engines with <class 'ipyparallel.cluster.launcher.MPIEngineSetLauncher'>
  0%
 0/256 [00:00<?, ?engine/s]

After Completing the Calculation¶

Starting 256 engines with <class 'ipyparallel.cluster.launcher.MPIEngineSetLauncher'>
100%
 256/256 [00:27<00:00,  2.64engine/s]
Hello World from rank 0. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 1. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 2. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 3. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 4. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 5. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 6. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 7. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 8. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 9. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 10. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 11. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 12. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 13. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 14. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 15. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 16. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 17. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 18. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 19. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 20. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 21. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 22. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 23. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 24. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 25. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 26. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 27. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 28. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 29. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 30. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 31. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 32. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 33. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 34. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 35. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 36. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 37. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 38. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 39. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 40. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 41. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 42. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 43. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 44. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 45. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 46. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 47. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 48. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 49. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 50. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 51. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 52. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 53. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 54. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 55. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 56. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 57. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 58. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 59. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 60. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 61. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 62. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 63. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 64. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 65. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 66. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 67. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 68. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 69. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 70. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 71. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 72. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 73. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 74. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 75. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 76. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 77. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 78. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 79. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 80. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 81. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 82. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 83. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 84. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 85. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 86. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 87. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 88. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 89. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 90. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 91. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 92. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 93. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 94. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 95. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 96. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 97. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 98. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 99. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 100. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 101. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 102. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 103. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 104. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 105. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 106. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 107. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 108. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 109. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 110. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 111. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 112. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 113. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 114. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 115. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 116. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 117. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 118. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 119. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 120. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 121. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 122. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 123. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 124. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 125. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 126. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 127. total ranks=256. host=cn090.karolina.it4i.cz
Hello World from rank 128. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 129. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 130. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 131. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 132. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 133. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 134. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 135. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 136. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 137. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 138. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 139. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 140. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 141. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 142. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 143. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 144. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 145. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 146. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 147. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 148. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 149. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 150. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 151. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 152. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 153. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 154. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 155. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 156. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 157. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 158. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 159. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 160. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 161. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 162. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 163. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 164. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 165. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 166. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 167. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 168. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 169. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 170. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 171. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 172. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 173. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 174. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 175. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 176. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 177. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 178. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 179. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 180. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 181. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 182. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 183. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 184. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 185. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 186. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 187. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 188. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 189. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 190. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 191. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 192. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 193. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 194. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 195. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 196. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 197. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 198. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 199. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 200. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 201. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 202. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 203. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 204. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 205. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 206. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 207. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 208. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 209. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 210. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 211. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 212. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 213. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 214. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 215. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 216. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 217. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 218. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 219. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 220. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 221. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 222. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 223. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 224. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 225. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 226. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 227. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 228. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 229. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 230. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 231. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 232. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 233. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 234. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 235. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 236. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 237. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 238. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 239. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 240. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 241. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 242. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 243. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 244. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 245. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 246. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 247. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 248. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 249. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 250. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 251. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 252. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 253. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 254. total ranks=256. host=cn091.karolina.it4i.cz
Hello World from rank 255. total ranks=256. host=cn091.karolina.it4i.cz
Stopping engine(s): 1733739848