Publications
Multimodal Proximity and Visuotactile sensing with a selectively transmissive soft membrane
Jessica Yin, Gregory M. Campbell, James Pikul, and Mark Yim. IEEE International Conference on Soft Robotics (RoboSoft) 2022. *Best Student Paper Award*
The most common sensing modalities found in a robot perception system are vision and touch, which together can provide global and highly localized data for manipulation. However, these sensing modalities often fail to adequately capture the behavior of target objects during the critical moments as they transition out of static, controlled contact with an end-effector to dynamic and uncontrolled motion. In this work, we present a novel multimodal visuotactile sensor that provides simultaneous visuotactile and proximity depth data. The sensor integrates an RGB camera and air pressure sensor to sense touch with an infrared time-of-flight (ToF) camera to sense proximity by leveraging a selectively transmissive soft membrane to enable the dual sensing modalities. We present the mechanical design, fabrication techniques, algorithm implementations, and evaluation of the sensor's tactile and proximity modalities. The sensor is demonstrated in three open-loop robotic tasks: approaching and contacting an object, catching, and throwing. The fusion of tactile and proximity data could be used to capture key information about a target object's transition behavior for sensor-based control in dynamic manipulation.
Wearable soft technologies for haptic feedback and sensing
Jessica Yin, Ronan Hinchet, Herbert Shea, and Carmel Majidi. Advanced Functional Materials, 2020.
Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio‐compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state‐of‐the‐art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on‐body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.
Closing the Loop with Liquid-Metal Sensing Skin for Autonomous Soft Robot Gripping
Jessica Yin, Tess Hellebrekers, and Carmel Majidi. IEEE International Conference on Soft Robotics (RoboSoft), 2020.
Soft robots are often limited in high-level decision making and feedback control due to a lack of multimodal sensing capabilities and material architectures that tightly integrate sensing and actuation. However, the recent development of elastic multimodal sensing skins has created the opportunity for closing the loop in soft robotic systems. In this work, we present a sensor-based finite state machine for a soft sensorized gripper mounted to a 4-degree-of-freedom robot arm. The soft gripper actuates between binary open and close states by activating shape-memory alloy springs, and contains proximity, pressure, and orientation sensors. The closed-loop control is demonstrated through scanning, grasping, and sorting tasks driven by sensor feedback. Using a time-of-flight distance sensor, the system can calculate the length, width, height, and center of mass of an object within a 60 mm x 25 mm workspace. With the time-of-flight distance sensor, pressure sensors, and inertial measurement unit, the system can detect and respond to external perturbations that interfere with the grasp, release, and transport of the object. The control strategy demonstrated in this paper can be expanded in the future to integrate basic autonomy in other soft robot systems.
Real-Time Visualization of Neural Network Training to Supplement Machine Learning Education
Michael You and Jessica Yin. IEEE Integrated Stem Education Conference, 2019.
In machine learning, neural networks have excelled at performing tasks at a high level with a simple and flexible implementation. Neural networks are particularly well-suited for novice programmers due to the availability of open-source libraries like TensorFlow and Caffe. However, novice programmers often neglect to learn beyond the black-box behaviors that these libraries provide. Introductory college students often lack the understanding of neural network internals, such as hidden layers and activation functions, and their interactions during training, which are crucial to efficiently solving more complex problems. Here, we present Omega 3 , a device that opens up the black-box of neural networks by visually representing how hidden layers behave during training in real-time. In addition, Omega 3 provides an engaging tactile and visual educational experience to students, and waives the requirement for a strong programming background in order to learn about neural networks. In this paper, we will discuss the fabrication and set-up of Omega 3 as well as evaluate and compare Omega 3 to traditional lecture-based learning.
Liquid Metal-Microelectronics Integration for a Sensorized Soft Robot Skin
Tess Hellebrekers, Kadri Bugra Ozutemiz, Jessica Yin, and Carmel Majidi. IEEE International Conference on Intelligent Robots and Systems (IROS), 2018.
Progress in soft robotics depends on the integration of electronics for sensing, power regulation, and signal processing. Commercially available microelectronics satisfy these functions and are small enough to preserve the natural mechanics of the host system. Here, we present a method for incorporating microelectronic sensors and integrated circuits (ICs) into the elastomeric skin of a soft robot. The thin stretchable skin contains various solid-state electronics for orientation, pressure, proximity, and temperature sensing, and a microprocessor. The components are connected by thin-film copper traces wetted with eutectic gallium indium (EGaIn), a room temperature liquid metal alloy that allows the circuit to maintain conductivity as it deforms under mechanical loading. In this paper, we characterize the function of the individual sensors in air and water, discuss the integration of the microelectronic skin with a shape-memory actuated soft gripper, and demonstrate the sensorized soft gripper in conjunction with a 4 degree-of-freedom (DOF) robot arm.