Designing robotic systems that can change their physical form factor as well as their compliance to adapt to environmental constraints remains a major conceptual and technical challenge. To address this, we introduce the Granulobot, a modular system that blurs the distinction between soft, modular, and swarm robotics. The system consists of gear-like units that each contain a single actuator such that units can self-assemble into larger, granular aggregates using magnetic coupling. These aggregates can reconfigure dynamically and also split into subsystems that might later recombine. Aggregates can self-organize into collective states with solid- and liquid-like properties, thus displaying widely differing compliance. These states can be perturbed locally via actuators or externally via mechanical feedback from the environment to produce adaptive shape shifting in a decentralized manner. This in turn can generate locomotion strategies adapted to different conditions. Aggregates can move over obstacles without using external sensors or coordinate to maintain a steady gait over different surfaces without electronic communication among units. The modular design highlights a physical, morphological form of control that advances the development of resilient robotic systems with the ability to morph and adapt to different functions and conditions.