Go down the steps and enter the asylum. It's very haunted and cursed at the moment, so stay alert. Move down the path carefully and drop down into the big room. Loot the chest in the corner to trigger an attack. Two wraiths will appear. They can do a lot of damage, but they're a little slow. You should be able to outmaneuver them, although it may be unnecessary. If you have specter oil, use it now. Your sword should cut through them much better, and stun them fairly well.
I found it fairly easy to just stay on the attack. Roll away from them after a few solid fast hits, and then keep up the harassment. If you get a chance, Quen doesn't hurt. Fast attacks are fine for now. Also try to focus on one. Once it's one-on-one, it's fairly easy.
When the fight's over, read the papers that you got to clear away the first fire and open a path deeper into the mansion. Once again, we're basically just moving along and around. It's a fairly linear path. Keep an eye out as you go for chests in the side rooms with other files in them. They're not vital, but it's good for the backstory. You'll find Gridley at the end of it, though he seems to be in a bit of a sad state at the moment. We need to find some sedatives.
This isn't actually that hard either. Just walk down a bit and swing into the room on the right. Cast a Quen on yourself, and then approach the glowing desk with the alchemy equipment on it. There will be two wraiths that will spawn and approach. If you really need new specter oil, go ahead and use it. You can always make more.
The tactics will be roughly the same though. Once they're dead, you can grab the sedatives from the desk and walk out. Talking to Gridley this time will reveal more of the story. After a bit of a conversation, and with a little persuasion, he'll reveal the whole truth. It seems like Rupert and Gridley were not very good men in their earlier years.
Don't actually follow Gridley, as he'll run back down the hallway to hide for a bit. You need to go to the end of the hallway down the newly opened path. You'll find the Nilfgaardian's ghost in this room, kneeling over his old body. If you talk to him, he'll ask that you honor his request for revenge. For now, just note it and walk back out to Gridley. He'll automatically take you back out to see Rupert.