Abstract

Project Jenkins explores how neural activity in the brain can be decoded into robotic movement and, conversely, how movement patterns can be used to generate synthetic neural data. Using real neural data recorded from motor and premotor cortex areas of a macaque monkey named Jenkins, we develop models for decoding (converting brain signals into robotic arm movements) and encoding (simulating brain activity corresponding to a given movement)

For the interface between the brain simulation and the physical world, we utilized Koch v1.1 leader and follower robotic arms. We developed an interactive web console that allows users to generate synthetic brain data from joystick movements in real time.

Our results are a step towards brain-controlled robotics, prosthetics, and enhancing normal motor function. By accurately modeling brain activity, we take a step toward flexible brain-computer interfaces that generalize beyond predefined movements.

Introduction

Synthetic neural data generation and neuroprosthetic devices are active areas of research, sparked by advances in neuroscience and robotics [22, 4, 2, 15]. These fields have significant implications for brain-computer interfaces, rehabilitation, and simulation of brain dynamics for downstream tasks or gaining new understanding of the underlying neural mechanisms.

In this project, which we call "Project Jenkins," we explore such decoding and encoding of neural data from a macaque monkey named Jenkins. We used a publicly available dataset [5] containing neural firing patterns from Jenkins' motor and premotor cortical areas during a center-outreach task.

Generating synthetic neural activity enables researchers to test and refine decoding models without requiring continuous access to live neural recordings [12, 16], while neuroprosthetic advancements [18, 20, 21, 9, 7, 3, 8, 17] rely on robust encoding techniques to translate brain signals into precise motor commands.

Our aim was two-fold (Figure 1, 2):

• Decoding: Translate neural spiking data into predicted velocities for a robotic arm.
• Encoding: Generate synthetic neural activity corresponding to an intended robotic movement.

With this paper, we publish the developed open-source tools for both synthetic neural data generation and neural decoding, enabling researchers to replicate our methods and build upon them.

Obtaining Jenkins' Data

Figure 3. A typical monkey reaching task. Figure adapted from [13]

The neural data used in our project came from the primary motor cortex (M1) and caudal portion of the dorsal premotor cortex (PMd) areas of the brain of a rhesus macaque monkey, Jenkins. The dataset was published by Mark Churchland and others in 2021 [5] and is publically available here.

Jenkins was trained on multiple reaching tasks, where the goal of the task is to press on dots that randomly light up on the screen in front of it. Every time the monkey completes a trial successfully, he is rewarded with fruit juice. In our project, we used data from a center-outreach task, where the monkey always starts out with its hand in the middle of the screen, and the dots light up in one of 8 positions (0°, 45°, 90°, 135°, 180°, 225°, 270°, or 310°). All in all, there are more than a dozen hours of brain recordings together with the tracking of the monkey arm as it pressed dots on the screen thousands of times.

Decoding: from Neural Data to Robot Movement

Decoding is where we convert monkey brain data into the movements of the robotic arm. Whenever Jenkins wants to move his arm, neurons in his brain activate, calculating the trajectory of his planned movement, and this activation signal is sent through his spinal cord into the muscles in the monkey's arm. This sequence of events in time means that the movement of the arm depends on the recent history of neural data, and our model should take this fact into account.

Feature Construction. To predict the monkey's hand movements from brain recordings, we split time into bins of 20 ms. For each bin, we recorded the total number of spikes from each of the 192 neurons in that time bin (Figure 4). This gives a vector $\mathbf{x}_t \in \mathbb{R}^{192}$ every 20 ms. We also recorded the average $x$ and $y$ velocity of Jenkins' hand (from motion-capture data) during that same bin, forming a velocity vector $\mathbf{v}_t=\left(v_x, v_y\right)$.

Figure 4. The decoding procedure.

Neural Network Decoding Model. We employ a simple MLP (multilayer perceptron) with two hidden layers (sizes 256 and 128) and ReLU nonlinearities.

Thus, at each time step $t$, we feed $\mathbf{x}_{t-49}, \ldots, \mathbf{x}_t$ into the MLP to predict the current velocity $\mathbf{v}_t$. Despite its simplicity, this architecture performed effectively, achieving $R^2 \approx 0.9$ on a held-out test set.

Driving the Robotic Arm. We implemented a follower robotic arm using the Koch v1.1 design [1]. The arm has six servo motors arranged in a chain, and its end-effector position is determined by the servo angles. We decode velocity from the neural data, integrate it to obtain Cartesian coordinates $(X, Y)$ using the exponentially moving average (EMA), and then apply inverse kinematics (via the ikpy library, [14]) to calculate servo angles for each motor. We defined a kinematic chain by specifying the servo motors' properties and the connecting link geometries. This chain configuration enables both forward kinematics (calculating end effector position from servo rotations) and inverse kinematics (determining required servo angles from desired coordinates). The EMA step is important to avoid the accumulation of errors over time.

This loop yields continuous control of the robot in real time:

Encoding: from Robot Movement to Neural Data

Figure 5. Generating synthetic neural data.

While the decoding problem was relatively straightforward, the encoding stage proved to be considerably more challenging. The encoding model is designed to generate neural spiking patterns given a sequence of arm velocities (or positions).

Closed-Loop Simulation Challenge. The key difficulty lies in closed-loop simulation: to continuously produce simulated neural data, the model needs to take its own past outputs as new inputs, to produce future predictions based on what it has already said before. So if a model is a little bit off every time, then the errors from its output get fed back into the model, producing even larger errors in the next output, and so on.

Small errors can accumulate over time, destabilizing the generated signals. After several time bins, these errors would often blow up to large spiking rates or collapse to near-zero activity. Success required fiddling with multiple different ways of formatting the inputs, finding the right architecture (transformer or LSTM) and training hyperparameters, as well as training for a long time ( 400 epochs).

Transformer-based Encoder. Since the present movement of the arm depends on past neural data, it follows that the present neural data depends on the future (planned) movement of the arm. That is, to generate neural data, the model needs to know what the future arm movement will be. Accordingly, we process the data so that for every time bin, we input all of the past brain activity to the model, as well as the future arm movement velocities (Figure 5). Specifically, we provided:

• Past neural activity: Binned spike counts from previous time steps.
• Future arm movement: A "look-ahead" window of 40 bins (800 ms) of velocities.

The model is trained to predict how many times each neuron will spike in the current 20 ms time bin. We formulate this problem as a 9-way classification: either a neuron will spike 0 times (be quiet), or $1, 2, \ldots, 7$, or $8+$ times.

The model was trained to output a 9-category distribution for each neuron, representing the number of spikes: $\{0, 1, \ldots, 7, 8+\}$. We train our encoding model to output what it thinks are the probabilities of each neuron's number of spikes being in any of our defined categories. This discrete classification formulation, akin to next-token prediction in language models [19], [11], helped stabilize training.

Training Procedure. We trained for approximately 400 epochs, carefully tuning hyperparameters (learning rate, batch size, dropout) to avoid divergence. We found best results by setting the learning rate to 0.0005 and no weight decay $(\lambda_{wd} = 0)$. The training took under 4 hours on an off-the-shelf GPU with 12GB of RAM. We experimented with an LSTM-based approach [10] but eventually found a transformer architecture to be more robust.

Closing the Loop: Robot Movement to Neural Data and Back to Movement

To accurately record the robotic arm's movement, we assembled a Koch v1.1 leader arm, sampling its $(x, y, z)$ coordinates at a frequency of 50 Hz. Since our study focuses exclusively on two-dimensional data, we discarded the $z$-coordinate. We then differentiated the $(x, y)$ positions to obtain the velocities $(V_x, V_y)$, which serve as input for our encoding model.

After the model generates neural data corresponding to the movement inputs, we apply the decoding procedure to transform this synthetic neural data back into velocity components ($V_x, V_y$). To reconstruct reliable spatial coordinates $(x, y)$ from these velocities, we employ an exponential decay filter ($(\lambda_{\text{decay}} = 0.95)$), which mitigates the compounding of integration errors over time.

Finally, the filtered positional data is passed to the Koch follower arm, where inverse kinematics algorithms compute the necessary servo rotations, enabling the follower arm to accurately replicate the original movements. The resulting system can be seen in Figure 2 and on the project website.

Interactive Interface

To enable users without direct robotic hardware experience to interactively generate neural data from movement, we developed an interactive web application that allows joystick manipulation. This app records the velocities from joystick input and processes them through our transformer model to produce synthetic neural data, visualizing the output in real time directly in the browser. Notably, the transformer model runs entirely within the user's browser rather than on a remote server. To achieve this, we converted our PyTorch model into a browser-compatible .onnx format using ONNX Runtime [6], ensuring efficient local execution.

Conclusion and Future Directions

Project Jenkins demonstrates the feasibility of translating real neural activity into robotic arm movement and generating synthetic neural data to accompany or predict such movement. While our decoding model was successful and robust, the encoding model required more intricate architectures (transformers) and careful training to produce stable spike patterns over time.

In practice, these methods can be extended to broader applications, such as human BCIs, prosthetics, and motor rehabilitation. Although our data was limited to eight primary reaching directions, preliminary experiments suggest generalization to more complex trajectories (e.g., drawing circles). Future work will focus on:

• Testing on extended movement repertoires and more complex tasks
• Improving the robustness of the encoding model in closed-loop simulation.
• Exploring real-time human-interface prototypes that adapt these neural decoders.

Citation

@misc{2025jenkins,
  title={Project Jenkins: Turning Monkey Neural Data into Robotic Arm Movement, and Back}, 
  author={Andrii Zahorodnii and Dima Yanovsky},
  year={2025},
  eprint={2503.14847},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2503.14847}, 
}