Search Documentation

Glenn, Oakley, Owens

Search our client documentation below, optionally filtered by one or more systems.

Oakley, Owens

Apache Spark is an open source cluster-computing framework originally developed in the AMPLab at University of California, Berkeley but was later donated to the Apache Software Foundation where it remains today. In contrast to Hadoop's disk-based analytics paradigm, Spark has multi-stage in-memory analytics. Spark can run programs upto 100x faster than Hadoop’s MapReduce in memory or 10x faster on disk. Spark support applications written in python, java, scala and R

Oakley, Owens, Ruby

Problem Description

Our current GPFS file system is a distributed process with significant interactions between the clients. As the compute nodes being GPFS flle system clients, a certain amount of memory of each node needs to be reserved for these interactions. As a result, the maximum physical memory of each node allowed to be used by users' jobs are reduced, in order to keep the healthy performance of the file system. In addition, using swap memory is not allowed anymore. 

Wednesday, October 5th

3:00 - 5:00 pm

Allocations Committee Meeting (members only)

6:00 - 7:30 pm

SUG Executive Meeting (members only)                           

Wednesday, October 5th

3:00 - 5:00 pm

Allocations Committee Meeting (members only)

6:00 - 7:30 pm

SUG Executive Meeting (members only)                           

Owens

Memory Limit:

It is strongly suggested to consider the memory use to the available per-core memory when users request OSC resources for their jobs. On Owens, it equates to 4GB/core or 124GB/node.

Ruby

Caffe is "a fast open framework for deep learning".

From their README:

Oakley, Owens

From WARP3D's webpage:

Owens

Here are the queues available on Owens. Please note that you will be routed to the appropriate queue based on your walltime and job size request.

Name Nodes available max walltime max job size notes

Serial

Pages