{"id":357,"date":"2021-03-09T12:43:52","date_gmt":"2021-03-09T12:43:52","guid":{"rendered":"https:\/\/www.embl.org\/about\/info\/it-services\/?page_id=357"},"modified":"2022-12-01T23:26:02","modified_gmt":"2022-12-01T23:26:02","slug":"efficient-computation","status":"publish","type":"page","link":"https:\/\/www.embl.org\/about\/info\/it-services\/efficient-computation\/","title":{"rendered":"Efficient computation"},"content":{"rendered":"\n
\n
\n\n
\"\"
Jaime Huerta Cepas<\/strong>
Research Scientist, Bork Group<\/a><\/strong><\/figcaption><\/figure>\n\n<\/div>\n<\/div>\n\n\n
\n
\n\n

I often use the HPC cluster for the projects I\u2019m working on. Recently we used the cluster to analyse around 2,000 genomes, identifying similar DNA sequences in different organisms \u2013 indicating that they have a common evolutionary ancestor. This generated roughly 1 million clusters of sequences, with each cluster needing to be analysed independently.<\/p>\n\n\n\n

I also use the cluster for functional annotation of sequences, which requires the same type of massive parallel computation. The HPC cluster is perfect for this, because we need a lot of CPU time in independent nodes, and we need to run as many parallel jobs as possible.<\/p>\n\n\n\n

If we have something like 1,000 jobs running at the same time and writing intensively to the disk, this can cause the file system to crash. So, if I need to run something like that, I usually get in contact with Jurij first. With him I can plan the best strategy to submit my jobs at any time, optimising the use of disk, CPU, and memory according to the current state of the cluster. For instance, we now have a very good system to automatically process data using scratch and temporary local disk areas, then move the results back to the shared file system. This has speeded up our computations a lot.<\/p>\n\n\n\n

I\u2019m very happy with the new system they\u2019re setting up, using modules and Slurm, which is the state-of-the-art way of doing things in high-performance computing. It\u2019s really appreciated by the people who are using these resources a lot, myself included.<\/p>\n\n\n\n

Another thing I like here is having direct access to the admins, so if something is failing I can just message them straight away. We have a chat system with a channel for the HPC cluster, and the admins are very responsive. That\u2019s very different to what you\u2019d have if using external HPC resources, which don\u2019t have such a personalised support system. In that case you\u2019d have to apply for computing time in an external cluster, and then the support you\u2019d get would be much more limited. Here I can design the jobs I\u2019m submitting directly with Jurij, and ask what he recommends. Being able to do that is very cool.<\/p>\n\n\n\n

<\/a><\/p>\n\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"embl_taxonomy":[],"class_list":["post-357","page","type-page","status-publish","hentry"],"acf":[],"embl_taxonomy_terms":[],"_links":{"self":[{"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/pages\/357","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/comments?post=357"}],"version-history":[{"count":3,"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/pages\/357\/revisions"}],"predecessor-version":[{"id":3534,"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/pages\/357\/revisions\/3534"}],"wp:attachment":[{"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/media?parent=357"}],"wp:term":[{"taxonomy":"embl_taxonomy","embeddable":true,"href":"https:\/\/www.embl.org\/about\/info\/it-services\/wp-json\/wp\/v2\/embl_taxonomy?post=357"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}